Synchronization of distributed cable modem network components

ABSTRACT

A distributed Cable Modem Termination System (CMTS) includes a head end, a downstream transmitter hub, and a plurality of cable modems that all establish frequency lock with a common frequency reference. The head end transmits a plurality of time stamps from the head end to the plurality of cable modems via a packet data network, the downstream transmitter hub, and cable modem network plant. Each of the plurality of cable modems performs smoothing operations on the plurality of time stamps to establish phase lock with the head end. The downstream transmitter and the plurality of cable modems perform ranging operations to establish phase lock among the plurality of cable modems. In an alternate operation, the frequency reference includes marker sequences that the devices of the distributed CMTS use to establish phase lock.

CROSS REFERENCES TO RELATED APPLICATIONS

This application is a continuation-in-part of U.S. Utility applicationSer. No. 11/031,568, filed Jan. 6, 2005 and is a continuation-in-part ofU.S. Utility application Ser. No. 11/061,147, filed Feb. 18, 2005, bothof which claim priority to U.S. Provisional Patent Application Ser. No.60/609,663, filed Sep. 14, 2004, to U.S. Provisional Patent ApplicationSer. No. 60/629,781, filed Nov. 20, 2004, and to U.S. Provisional PatentApplication Ser. No. 60/635,531, filed Dec. 11, 2004, all of which areincorporated herein by reference for all purposes.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The invention relates generally to communication systems and, moreparticularly, it relates to cable modem communication systems.

2. Description of Related Art

Conventional cable modem communication systems include Cable ModemTermination Systems (CMTSs), serviced Cable Modems (CMs), and a cablemodem network plant, i.e., hybrid fiber-coaxial media thatcommunicatively couples these devices. The CMTS services datacommunications for the CMs via downstream transmissions from the CMTS tothe CMs and upstream transmissions from the CMs to the CMTS. The DataOver Cable Service Interface Specification (DOCSIS) typically governsthe transmission and receipt of signals of the cable modem communicationsystem. In its various forms, DOCSIS supports Time Division MultipleAccess (TDMA), Frequency Division Multiple Access (FDMA), and CodeDivision Multiple Access (CDMA) operations. Ranging and registeringoperations are performed to manage the timing of communications betweenthe CMTS and the CMs.

The structure of CM communications systems continues to evolve. Oneevolution of the structure of cable modem communication systems includesdistributing the CMTS across differing devices that intercouple via apacket data network. Operation of the CM communication system requiressynchronization of the distributed CMTS components. Because the packetdata network introduces significant jitter, it would likely beimpractical to use the packet data network to meet the accuracy requiredfor synchronization of CMTS components without adding excessive latency.Thus, a need exists for synchronizing the distributed CMTS componentswhile meeting all system requirements.

BRIEF SUMMARY OF THE INVENTION

The present invention is directed to apparatus and methods of operationthat are further described in the following Brief Description of theDrawings, the Detailed Description of the Invention, and the claims.Other features and advantages of the present invention will becomeapparent from the following detailed description of the invention madewith reference to the accompanying drawings.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

FIG. 1 is a system diagram illustrating a cable modem communicationsystem having distributed Cable Modem Termination System (CMTS) head endcomponents;

FIG. 2 is a system diagram illustrating a cable modem communicationsystem constructed according to the present invention having distributedCMTS components inter-coupled by a packet switched network;

FIG. 3 is a flow chart illustrating operation according to an embodimentof the present invention;

FIG. 4 is a flow chart illustrating a first embodiment of the presentinvention for synchronizing clocks of a downstream transmitter hub andan upstream receiver hub;

FIG. 5 is a block diagram illustrating an embodiment of the presentinvention for establishing packet flow synchronization between a headend and a downstream transmitter hub;

FIG. 6 is a flow chart illustrating another embodiment establishingpacket flow synchronization between a head end and an upstream receiverhub according to the present invention;

FIG. 7 is a flow chart illustrating operation according to anotherembodiment of the present invention for synchronizing CMTS components;

FIG. 8 is a flow chart illustrating operation according to still anotherembodiment of the present invention for synchronizing distributed CMTScomponents;

FIG. 9 is a system diagram used to describe operations of packet datanetwork elements according to the present invention;

FIG. 10 is a flow chart illustrating operations of packet data networkelements according to the present invention;

FIG. 11 is a system diagram illustrating a cable modem communicationsystem constructed according to another embodiment of the presentinvention having distributed CMTS components inter-coupled by a packetswitched network;

FIG. 12 is a flow chart illustrating distributed CMTS componentsynchronization operations according to an embodiment of the presentinvention; and

FIG. 13 is a flow chart illustrating distributed CMTS componentsynchronization operations according to another embodiment of thepresent invention.

DETAILED DESCRIPTION OF THE INVENTION

FIG. 1 is a system diagram illustrating a cable modem communicationsystem having distributed Cable Modem Termination System (CMTS) head endcomponents. As is illustrated in FIG. 1, the distributed CMTS includes ahead end 102 and distributed hubs 108. Each of the distributed hubs 108services a respective Data Over Cable System Interface Specification(DOCSIS) Physical Layer (PHY) domain 104 and 106. Each DOCSIS PHY domain104 and 106 services a plurality of cable modems (CMs) 110. The head end102 couples to the hubs 108 via a media 114. The head end 102 transfersdata to, and receives data from the cable modems 100 via the hubs 108and the media 114.

Synchronization of transmissions within the PHY domains 104 and 106 is arequirement. When supporting TDMA operations, burst transmissions (bothupstream and downstream) within each DOCSIS PHY domain 104 and 106 mustbe sent at accurate times (100 ns level) to avoid collision. Whenoperating in the DOCSIS 2.0 S-CDMA mode, timing is even more criticalbecause synchronization to the symbol level is required. Thus, a morestringent synchronization on the order of 1 ns is required. In order tomeet these timing requirements, the head end 102 could maintain areference clock, e.g., operating at 10.24 MHz. The hubs 108 could thenlock their clocks to the reference clock of the head end 102 via link114. Each CM 110 could then lock its clock to the downstream symbolclock, which is synchronized to the 10.24 MHz reference clock. Rangingoperations could then be employed to adjust the offsets of the clocks ofthe CMs 110 to within 1 ns, typically. Once ranged, a CM 110 should notdrift off in time, due to the closed-loop synchronous aspects of thesystem. However, to allow for changes in the cable modem network plantpropagation delay as temperature, wind and other factors that vary overtime, ranging is repeated every 30 seconds, or more often, for each CM110. With this synchronization enacted, the entire DOCSIS PHY 104 and/or108 are synchronous.

FIG. 2 is a system diagram illustrating a cable modem communicationsystem constructed according to the present invention having distributedCMTS components inter-coupled by a packet data network 202. Adistributed CMTS includes a head end 102 and a hub 108 that includes adownstream transmitter hub 214 and an upstream receiver hub 216. Thehead end 102, the downstream transmitter hub 214, and the upstreamreceiver hub 216 couple to one another via the packet data network 202.The packet data network 202 may be an Ethernet network or another typeof packet data network. The downstream transmitter hub 214 and theupstream receiver hub 216 of the hub 108 may reside in differingfacilities. However, in other embodiment the downstream transmitter hub214 and the upstream receiver hub 216 may be located in a singlefacility.

The downstream transmitter hub 214 and the upstream receiver hub 216couple to cable modem network plant 210. CMs 110 also couple to cablemodem network plant 210. The cable modem network plant 210 may be ahybrid fiber coaxial cable modem network 210 or another type of cablemodem network plant that's generally known. The distributed CMTSservices data communications between data network 112 and CMs 110 viathe cable modem network plant 210.

The distributed CMTS operates according to the DOCSIS specification, insome embodiments. As was discussed above, the DOCSIS MAC uses TDMA,FDMA, and/or S-CDMA to service data communications with the CMs 110across the cable modem network plant 210. Thus, according to thesespecifications, it is desirable for the head end 102 to be substantiallyin synchronization with the downstream transmitter hub 214 and theupstream receiver hub 216 from a packet data flow perspective. Thedownstream transmitter hub 214 includes a clock (timing base) 206. Theupstream receiver hub includes clock (timing base) 208. The clocks(timing bases) 206, and 208 of the downstream transmitter hub 214 andthe upstream receiver hub 216, respectively, should be sufficientlysynchronized to satisfy the timing requirements of the DOCSISspecification. Thus, according to the present invention, the clocks 206and 208 of the distributed CMTS components are sufficiently synchronizedso that the timing requirements of the DOCSIS standards are met. Theterms “clocks” and “timing bases” are used interchangeably herein whenreferring to mechanisms within the downstream transmitter hub 214 andthe upstream receiver hub 216 for keeping a system time or other timereference.

The packet data network 202 has a limited ability to distribute accuratetiming information due to its inherent jitter. The head end 102 mayinclude a clock (timing base) 204. One possibility for maintainingsystem time in the distributed CMTS structure of FIG. 2 is to transfertiming information from the head end 102 to the downstream transmitterhub 214 and the upstream receiver hub 216 over the packet data network202 using time stamps. The downstream transmitter hub 214 and theupstream receiver hub 216 would smooth the jitter caused by the packetdata network 202 and recover the timing base clock from the timestamps,typically employing a filter to smooth the time stamps. In order toconsider whether the structure of FIG. 2 can support such operations, ananalysis may assume the following typical parameters:

-   -   1. The packet data network 202 supports Gigabit Ethernet        operations.    -   2. Jitter on a Gigabit Ethernet link=1 ms.    -   3. After smoothing, timestamp jitter=100 ns, in order to meet        overall budget of 500 ns.    -   4. Timestamp messages are sent at a rate of 100 per second.

The smoothing that would be necessary is the ratio 1 ms/100 ns=1×10⁴.Assuming independent jitter on each timestamp, N=1×10⁸ timestamps mayhave to be averaged to achieve smoothing of 1×10⁴. This would require asmoothing time constant of 1×10⁸ sec/100=1×10⁶ sec, or 11.6 days, whichis impractical. In order to use the packet data network 202 todistribute timing information, the packet data network would have tohave a jitter limit of approximately 17 us, which is not currentlyfeasible.

Thus, according to the present invention, other techniques are employedto synchronize the components of the distributed CMTS. According to afirst aspect of the present invention, packet data flow from the headend 102 to the downstream transmitter hub 214 is substantiallysynchronized in an attempt to avoid overflow and underflow of a transmitbuffer of the downstream transmitter hub 214. These operations will bedescribed with reference to FIG. 6. Alternately, packet data flow fromthe head end 102 to the downstream transmitter hub 214 may besubstantially synchronize using a control loop within the downstreamtransmitter hub 214 stimulated by time stamps transmitted from the headend 102. These operations will be described further with reference toFIG. 5.

In order to complete synchronization of the distributed CMTS components,the clocks 206 and 208 of the downstream transmitter hub 214 and theupstream receiver hub 216, respectively, are substantially synchronized.Using a first technique according to the present invention, suchsynchronization is performed using ranging operations supported by atleast one cable modem communicatively coupled to both the upstreamreceiver hub 216 and the downstream transmitter hub 214 via cable modemnetwork plant 210. These operations are described further with referenceto FIGS. 3 thru 4. Another technique for synchronizing the clock 208 ofthe upstream receiver hub 216 to the clock 206 of the downstreamtransmitter hub includes detecting clock drift based upon changes inalignment of symbols received by the upstream receiver hub 216 whensupporting S-CDMA operations. This technique will be described furtherwith reference to FIG. 7.

According to another embodiment of the present invention, the head end102 transmits a plurality of data packets to the downstream transmitter214 via the coupling packet data network 202. Each data packet is markedwith a “measured delay” Quality of Service (QOS). Each data packetincludes a time stamp that is based upon a clock of the head end and adelay tag.

Operation continues with the downstream transmitter hub 214 receivingthe plurality of data packets via the packet data network 202. Then, thedownstream transmitter hub 214 smoothes the time stamps of the pluralityof data packets based upon corresponding delay tags of the data packets.The delay tag of each data packet that is received by the downstreamtransmitter hub 214 is representative of a corresponding delayencountered in traversing the packet data network 202. Thus, the delaytag represents a delay introduced by the packet data network 202 for theparticular data packet. The smoothing of the plurality of time stamps bythe downstream transmitter hub 214 may include the downstreamtransmitter hub 214 first adjusting the time stamps based upon thecorresponding delay tags. Then, based upon the smoothed time stamps, thedownstream transmitter hub 214 synchronizes its clock 206 with the clock204 of the head end 102.

Finally, operation concludes with synchronizing a clock 208 of theupstream receiver hub 216 with the clock 206 of the downstreamtransmitter hub 214. In synchronizing the clock 208 of the upstreamreceiver hub 216 with the clock 206 of the downstream transmitter hub214, ranging operations supported by at least one cable modem 110 may beemployed. Alternately, a direct connection between the downstreamtransmitter hub 214 and the upstream receiver hub 216 may be employed.These operations will be described further with reference to FIGS. 8-10.

According to an alternate embodiment of the present invention, the datapackets that include the measured delay QOS marking, the time stamp, andthe delay tags are transmitted from the head end 102 to the upstreamreceiver hub 216. In such case, the upstream receiver hub 216 performssmoothing operations of the time stamps to synchronize its clock 208 tothe clock 206 of the head end 102. Then, the clock 206 of the downstreamtransmitter hub 214 is synchronized to the clock 208 of the upstreamreceiver hub 216. The previously described operations may be employedwith this embodiment as well for such clock synchronization. Theseoperations will also be described further with reference to FIGS. 8-10.

FIG. 3 is a flow chart illustrating operation according to an embodimentof the present invention. Operation commences in initializing the headend 102, the downstream transmitter hub 214, and the upstream receiverhub 216 for operation (step 302). With this step complete, operationcontinues in establishing packet flow synchronization between the headend 102 and the downstream transmitter hub 214 (step 304). Techniquesfor performing packet flow synchronization are described further withreference to FIGS. 5 and 6. These operations may further be describedherein as regulating packet data flow from the head end 102 to thedownstream transmitter hub 214.

Operation continues with synchronizing the clock 206 of the downstreamtransmitter hub 214 and the clock 208 of upstream receiver hub 216 (step306). According to the present invention, synchronizing the clock 208 ofthe upstream receiver hub 216 with the clock 206 of the downstreamtransmitter hub 206 is performed using ranging operations supported byat least one CM 110 coupled to both the upstream receiver hub 216 and tothe downstream transmitter hub 214 via the cable modem network plant210.

According to one embodiment of step 306, the ranging operations aresupported by a plurality of CMs 110 for which the head end 102, thedownstream transmitter hub 214, and the upstream receiver hub 216together provide data service. Normally, ranging is done every 15seconds or so for each CM 110, so tens to hundreds of time-of-arrivalmeasurements are made each second. With another embodiment, the rangingoperations are supported by a dedicated clock synchronization CM. Thededicated clock synchronization CM may be ranged at a high rate, perhaps10 to 100 times per second. These additional ranging measurements, whensmoothed (in a control loop with a slow time constant, say one second orseveral seconds), give a high confidence in the estimation of the timingdrift between the clock 206 of the downstream transmitter hub 214 andthe clock 208 of the upstream receiver hub 216. However, since the delayof the cable plant is assumed not to be changing quickly and thefrequencies of the clocks should not be changing except due to explicitand intentional changes plus natural clock drift, it should not benecessary to combine very large numbers of measurements to obtain atiming estimate with sufficient accuracy and confidence. Of course,combinations of these embodiments may be employed using both normalranging information and measurements of one or more dedicated clocksynchronization CMs. In either case, the ranging operations are employedto synchronize the clocks 206 and 208 of the downstream transmitter hub214 and the upstream receiver hub 216, respectively. Also, data burstsmay be used in addition to ranging bursts to glean additionalinformation; if data bursts arrive early or late at the receiver hub,that can help define a trend, which implies a frequency offset in thereference clocks, which can then be trimmed out.

Operation of FIG. 3 continues in servicing the CMs, according to one ormore aspects of the DOCSIS specification (step 308). The operations ofstep 308 continue until it is determined that adjustment of the clock206 of the downstream transmitter hub 214 and/or of the clock 208 of theupstream receiver hub is required (step 310). When adjustment of theclocks 206 and 208 of the downstream transmitter hub 214 and of theupstream receiver hub 216, respectively (as determined at step 310)is/are required, operation returns to step 306. Otherwise, operationproceeds to step 312 where it is determined whether adjustment of thepacket flow synchronization of the head end 102 with the downstreamtransmitter hub 214 is required. If so operation returns from step 312to step 304. If not, operation proceeds from step 312 to step 308 whereservice of the CM continues.

As the reader will appreciate, the operations of FIG. 3 may be performedin varying orders that may differ from the order of operations presentedin FIG. 3. Synchronization of the clocks 206 and 204 of the downstreamtransmitter hub 214 and the head end 102, respectively, may occur duringthe packet flow synchronization of the head end with the downstreamtransmitter hub 214 and/or during the time period the CMs are serviced.Typically, maintaining synchronization, both packet flow synchronizationand clock synchronization is ongoing and may be performed at any time.

FIG. 4 is a flow chart illustrating a first embodiment of the presentinvention for synchronizing clocks of a downstream transmitter hub andan upstream receiver hub (step 306 of FIG. 3). Operation commences bytransmitting ranging bursts from the downstream transmitter hub 214 to adedicated clock synchronization CM or to one or a plurality of CMs (step402). The dedicated clock synchronization CM or the one or a pluralityof CMs receive the ranging bursts from the downstream transmitter hub214, process the ranging bursts, and transmit the ranging bursts to theupstream receiver hub 216 (step 404). The upstream receiver hub 216receives the ranging bursts and analyzes the ranging bursts to estimatethe clock frequency difference between its clock 208 and the clock 206of the downstream transmitter hub 214 (step 406). The clock drift maymanifest itself over time such that adjustments of the upstream receiverhub 216 clock 208 or the downstream transmitter hub 214 clock 206 may beperformed at a regular interval, at a non-regular interval, or when thedrift exceeds a threshold. Thus, the upstream receiver hub 216 adjustsits clock 208 if necessary or alternatively directs the downstreamtransmitter hub 214 to adjust its clock 206, if necessary (step 408).

FIG. 5 is a block diagram illustrating an embodiment of the presentinvention for establishing packet flow synchronization between a headend and a downstream transmitter hub. Shown in FIG. 5 are the head end102, the clock 204 of the head end 102, the downstream transmitter hub214, the clock 206 of the downstream transmitter hub 214, and the packetdata network 202. According to the embodiment of step 304 of FIG. 3 acontrol loop 502 may be employed to establish packet flowsynchronization between the head end 102 and the downstream transmitterhub 214. In such embodiment, the control loop 502 is associated with thedownstream transmitter hub 206 and is responsive to timing information(time stamps 504) of the head end 102 clock 204 received across thepacket data network 202. Control loops are generally known and will notbe described further herein except as how they relate to the presentinvention. Because the packet data network 202 is jittery, the controlloop 502 may be employed to synchronize the head end clock 204 with thedownstream transmitter hub clock 214 over time with adequate precisionfor some supported operations, e.g., 100 ns required for TDMA operation.The latency introduced by the control loop 502 may be excessive for someapplications. Further, the time to lock the control loop 502 may beexcessive for some applications. Thus, while the control loop 502 is notsufficient to synchronize the head end clock 204 with the downstreamtransmitter hub clock 206 in all cases, the control loop 502 istypically sufficient to establish packet flow synchronization betweenthe head end 102 and the downstream transmitter hub 214.

Using the control loop 502, the head end 102 transmits time stamps (TS)504 with packet data. The TS 504 indicate instantaneous values of theclock (time base) 204 of the head end 102, the time base being driven bythe clock 204. The control loop 502 of the downstream transmitter hub214 compares the values of TS 504 at the time they are received with thetransmitter hub's 214 own clock (timing base) 206. The downstreamtransmitter hub 214 clock (timing base) 206 may be initialized to thevalue of the first TS 504 when the first TS 504 is received, andsubsequently operated in a closed loop with the clock (timing base) 204of the head end 102 by comparing subsequent TS 504 with the timing base206. The result of the comparison between a received TS 504 and thetiming base 206 can be filtered and scaled to give desired loopperformance characteristics. Typically the filter is a form of low passfilter, either an FIR filter or IIR filter. The filter may resemble anintegrator, although the filter response may be somewhat different froma pure integrator (another zero and pole) in order to ensure loopstability. The scaling of the filter result, i.e. multiplication by aconstant, can control the loop gain. Such loops can be designed for highgain i.e. quick response or low gain i.e. slow response. The loop can bedesigned for variable gain i.e. variable response, so as to achieve lockwith the head end 102 clock 204 frequency quickly using high gain andalso to achieve very low variations in transmitter hub 214 clockfrequency once locked has been achieved, or partially achieved. Therecan be any number of gain settings. Loop design, gain settings, locksdetection, etc. are well known in PLL design.

FIG. 6 is a flow chart illustrating another embodiment for establishingpacket flow synchronization between a head end and an upstream receiverhub according to the present invention. The operations of FIG. 6correspond to steps 304 and 308 of FIG. 3. Operation commences with,based upon clock frequency difference specifications of the head end 102and of the downstream transmitter hub 214, determining a packet dataflow rate that should prevent overflow of a transmit buffer of thedownstream transmitter hub 214 (step 602). The operations of step 602would typically be performed in a design process and be laterimplemented. Operation continues with the head end 102 transmittingpacket data from the head end at a rate that that do not exceed thepacket data flow rate (step 604). The downstream transmitter hub 214receives the packet data, forms the packet data into data frames, andtransmits the data frames to the CMs 110 according to the DOCSISspecification, for example. At some time, if the downstream transmitterhub 214 may be transmitting packet data at a greater rate than it isreceiving packet data from the head end 102, downstream transmitter hub214 transmit buffer underflow may occur. Thus, at step 606 thedownstream transmitter hub 214 determines whether a downstreamtransmitter hub 214 transmit buffer underflow has occurred or is aboutto occur, i.e. if the transmit buffer becomes empty. If a downstreamtransmitter hub buffer becomes empty, or if it is about to go empty, thedownstream transmitter hub transmits one or more null data frames tocure the underflow situation (step 608), i.e. until there is sufficientdata in the transmit buffer to enable resumption of transmission ofdata. However, during normal operations with the head end 102substantially synchronized with the downstream transmitter hub 214 on apacket flow basis, the downstream transmitter hub 214 forms andtransmits data frames carrying packet data to serviced CMs withoutinsertion of null frames (step 610).

The relationship of the data rate demand from the head end 102 to thedownstream transmitter hub 214 data rate, and the prevalence of nullpackets are design variables with multiple solutions. In scenarios wherethe network delay (between the head end 102 and the downstreamtransmitter hub 214) is bounded to a known upper limit, the head end 102data rate can be almost equal to the downstream transmitter hub 214rate, with the required difference being determined by the maximumdifference between the clock rates of the head end 102 and thedownstream transmitter hub 214. If for example each clock has a worstcase tolerance of 5 ppm, then the total worst case difference is 10 ppm,and in this case the head end 102 should limit its bandwidth demand to1-10 ppm=0.9999 of the nominal transmitter hub 214 data rate, to coverthe case where the head end 102 is at the max frequency and thedownstream transmitter hub 214 is at the minimum frequency. Assuming adata rate of 38 Mbps and that a null data frame consists of 188 byte(1504 bit) data packets, for example, the downstream transmitter hub 214would insert null data frames at a rate of 10 ppm*38 Mbps/1504bits/packet ˜=0.25 data frames/second or about one null data frame per 4seconds. Of course, this rate depends on the actual relative frequenciesof the head end and hub and is not independent of that relationship. Ina case where the head end 102 is operating at a minimum specifiedfrequency and the transmitter hub 214 is operating at a maximumspecified frequency, the rate of inserted null data frames is doubled toabout 0.5 packets/second. If the head end 102 is operating at maximumspecified frequency and the transmitter hub 214 is operating at aminimum specified frequency, the same rate of about 0.5 packets/secondof null data frame insertion results.

There are various ways for the downstream transmitter hub 214 todetermine when to insert null data frames. In one approach, thedownstream transmitter hub 214 receives data packets from the head end102, frames it, and transmits it as soon as possible. Whenever thetransmitter hub 214 does not have enough data in its input buffer to beable to form a complete data frame, it sends a null data frame. In sucha design, the delay of data traversing the receive buffer trends towardszero. The delay can increase when the network delay changes from arelatively long delay to a relatively short delay. As long as thedownstream transmitter hub 214 data rate is greater than the head end102 data rate, the transmitter hub 214 will normally remove data fromthe buffer faster than data enters the buffer, and so the bufferfullness tends towards zero over the long term.

If there are multiple classes of data such that some data requires ashorter delay or more predictable delay, data can be transmitted fromthe head end 102 to the downstream transmitter hub 214 with suchclassifications indicated. The downstream transmitter hub 214 canprioritize the transmission of data according to the classification ofdata type, e.g. transmitting higher priority data in its buffer beforetransmitting lower priority data. This minimizes the additional delayimposed by the buffer on the more delay-sensitive data.

FIG. 7 is a flow chart illustrating operation according to anotherembodiment of the present invention for synchronizing CMTS components.Operation commences in initializing the head end 102, the downstreamtransmitter hub 214, and the upstream receiver hub 216 for operation(step 702). With this step complete, operation continues in establishingpacket flow synchronization between the head end 102 and the downstreamtransmitter hub 214 (step 704). Techniques for performing packet flowsynchronization were described with reference to FIGS. 5 and 6. Theseoperations may further be described herein as regulating packet dataflow from the head end 102 to the downstream transmitter hub 214.

Operation continues with the downstream transmitter hub 214 transmittingdownstream Code Division Multiple Access (CDMA) communications to aplurality of serviced cable modems 110 (step 706), e.g., downstreamtransmissions. Then, operation proceeds with the upstream receiver hubreceiving upstream CDMA communications from the plurality of servicedcable modems (step 708). The upstream receiver hub then determines adrift in alignment, if any, of symbols of the upstream communicationsreceived from the plurality of serviced cable modems with respect to atiming base of the upstream receiver hub (step 710). When the drift inalignment exceeds a threshold, as determined at step 712 to indicatethat the clock of the upstream receiver hub is no longer substantiallyaligned with the clock of the downstream transmitter hub, the clock ofthe upstream receiver clock is adjusted (Step 714). If no adjustment isrequired, as determined at step 712, operation proceeds to step 716where it is determined whether adjustment of the packet flowsynchronization of the head end 102 with the downstream transmitter hub214 is required. If so operation returns from step 712 to step 704. Ifnot, operation proceeds from step 712 to step 708 where service of theCM continues. Operation from step 714 also proceeds to step 706.Alternately, operation could proceed from step 714 to step 716.

As the reader will appreciate, the operations of FIG. 7 may be performedin varying orders that may differ from the order of operations presentedin FIG. 7. Typically, maintaining synchronization, both packet flowsynchronization and clock synchronization is ongoing and may beperformed at any time.

FIG. 8 is a flow chart illustrating operation according to still anotherembodiment of the present invention for synchronizing distributed CMTScomponents. Operation 800 commences with the initialization of the headend, the downstream transmitter hub, and the upstream receiver hub ofthe distributed CMTS (Step 802). Operation continues with the head endtransmitting a plurality of data packets to the downstream transmitterhub via a packet data network (Step 804). Each data packet is markedwith a measured delay Quality of Service (QoS) indicator and includes atime stamp that is based upon a clock of the head end. Each of thesedata packets also includes a delay tag. Operation continues with thedownstream transmitter hub receiving the data packets via the packetdata network with delay tags that have been adjusted by the packet datanetwork based upon latency in traversing the packet data network (Step806). As will be further described with reference to FIGS. 9 and 10, thedelay tags are updated by packet data network elements to indicate thedelay introduced into the transmission of the data packets as theytraverse the packet data network. The downstream transmitter hub adjuststhe time stamps of the plurality of data packets based uponcorresponding delay tags and then smoothes the time stamps (Step 808).

Operation continues with the downstream transmitter hub synchronizingits clock with the clock of the head end based upon the smoothed timestamps (Step 810). Then, the upstream receiver hub synchronizes itsclock with a clock of the downstream transmitter hub (Step 812). Suchsynchronization may include using ranging operations as is previouslydescribed herein. With synchronization complete, the distributed CMTScomponents to service a plurality of cable modems (Step 814). Fromtime-to-time, or periodically, resynchronization of the distributed CMTScomponents is required, as determined at Step 816. When suchdetermination is required, operation returns to Step 804. Whenresynchronization is not required, the operation returns to Step 814. Asthe reader will appreciate, these synchronization operations maycontinue without intervening delay during the normal operations of thedistributed CMTS. In such case, the head end regularly transmits datapackets containing time stamps and delay to the downstream transmitterhub. The downstream transmitter hub periodically, or continually adjuststhe time stamps and smoothes them to synchronization its clock with theclock of the head end.

According to still another embodiment of the present invention, the headend transmits data packets containing time stamps and delay tags to theupstream receiver hub. In such case, the operations of FIG. 8 would beemployed with the upstream receiver hub receiving the data packets withthe time stamps and delay tags. Then, the upstream receiver hub adjuststhe time stamps based upon the delay tags and smoothes the time stamps.The upstream receiver hub then synchronizes its clock with the clock ofthe head end based upon the smoothing. Then, the upstream receiver clocksynchronizes the clock of the downstream transmitter hub clock to itsclock via interaction with the downstream transmitter hub.

FIG. 9 is a system diagram used to describe operations of packet datanetwork elements according to the present invention. As shown in FIG. 9,a servicing packet data network 202 includes a plurality of packet datanetwork components 902-912, e.g., switch, router, hub, etc. As istypically the case in a packet data network, data packets traversing thenetwork may take different paths while traversing the packet datanetwork 202. Thus, according to the present invention, the delay tags ofthe data packets are modified by the packet data network elements toindicate the particular delay incurred by each particular packet datanetwork element in a servicing path. For example, some data packets maytraverse the packet data network 202 via path 916 that includes packetdata network elements 902, 908, 910, 912, and 906 before reaching thedownstream transmitter hub 214. In such case, each of the packet datanetwork elements 902, 908, 910, 912, and 906 adjusts the delay tags ofcorresponding data packets to indicate the delay introduced by theelement.

Comparatively, some data packets transmitted by head end 102 andreceived by downstream transmitter hub 214 will traverse the packet datanetwork 202 via path 914 that includes packet data network elements 902,904, and 906. As is shown, data packets traversing paths 916 and 914will likely have consistently different times of traversal. Thus,according to the present invention, the time stamps contained withinthese data packets may be adjusted using delay tags. By doing this, thedownstream transmitter hub 214 will be able to reduce the overall jitterof the plurality of data packets and more easily synchronize its clock206 with the clock 204 of the head end 204 using the adjusted timestamps.

For example, a data packet containing a time stack may have to wait in aqueue of packet data network element 904 while a larger packet (as largeas 1500 bytes, or perhaps even larger if concatenation is used) isprocessed by the packet data network element 904. The packet datanetwork elements 902-912 are designed so that they record the delayundergone by the data packet containing the time stamp and adjust thedelay tag with this information. The packet data network elements902-912 may use a simple counter to count increments of time (e.g.,nanoseconds or picoseconds) while the data packet containing the timestamp is in the queue. As the data packet containing the time stampmoves from packet data network element to packet data network element,it incurs various random delays. Because each delay is measured by theservicing packet data network element, and such delay is added to thedelay tag of the data packet, by the time the data packet has reachedthe final destination, e.g., the downstream transmitter hub 214, thedelay tag will have accumulated the sum of all the delays it incurred intraversing the packet data network 202.

The downstream transmitter hub 214 (or upstream receiver hub 216),processes the time stamps taking the delay tag information into accountso that it compensates for the random delay through the packet datanetwork 202. One way to take advantage of this delay tag is simply toadd the delay tag to the time stamp itself. For example, say the timestamp originally had the value t1. As it traverses the network, itencounters various delays which total t2, which are accumulated in thedelay tag. The downstream transmitter hub 214 then computes a correctedtime stamp value as t3=t1+t2. Such computation acts to remove the effectof the random jitter from the time stamp. It is as if the time stamp hadbeen emitted from the head end 102 at a later time, t3, and hadtraversed a packet data network 202 with zero delay, and essentiallyzero jitter.

FIG. 10 is a flow chart illustrating operations of packet data networkelements according to the present invention. Each of the packet datanetwork elements, 902-912 of FIG. 9, for example, may alter the delaytags of data packets having a measured delay QOS indicator. The packetdata network element receives a data packet (Step 1002). The packet datanetwork element then looks at the data packet to determine if the QOSfor the particular data packet is the measured delay QOS (Step 1004). Ifthe QOS of the data packet is not the measured delay QOS, operationends. However, if the QOS is the measured delay QOS, operation proceedsto Step 1006 wherein the packet data network element determines thedelay it introduces in receiving, servicing and transmitting the datapacket. Because the packet data network element may be relatively busyor relatively lightly loaded during any particular time interval, thedelay introduced by the packet data network element will vary fromtime-to-time. Thus, after determining this introduced delay, the packetdata network element adjusts the delay tag of the data packetaccordingly (Step 1008). The packet data network element then passes thedata packet along to a next packet data network element or to thedownstream transmitter hub 214 or upstream receiver hub 216. Theoperations of FIG. 10 apply to both the transmission of data packetsfrom the head end to the downstream transmitter hub and to thetransmission of data packets from the head end to the upstream receiverhub.

FIG. 11 is a system diagram illustrating a cable modem communicationsystem constructed according to another embodiment of the presentinvention having distributed CMTS components inter-coupled by a packetdata network 202. The components of FIG. 11 are same/similar to thosedescribed with reference to FIG. 2 and common numbering is maintained.As contrasted to the embodiment of FIG. 2, with the embodiment of FIG.11, a frequency reference is distributed from a server or a reference tothe head end 102, to the downstream transmitter hub 108, and to the CMs110. As an example, the frequency reference may be 10.24 MHz clock, asis specified in DOCSIS. The CMs 110 are each equipped with a PLL whichtrack the frequency reference by establishing frequency lock with thefrequency reference. Even after achieving the frequency lock of thefrequency reference, respective counters that keep time at the CMs 110have not yet been synchronized. The operations of FIG. 12 represent oneembodiment for performing counter synchronization of the CMTS componentswhile the operations of FIG. 13 represent another embodiment forperforming counter synchronization of the CMTS components.

FIG. 12 is a flow chart illustrating distributed CMTS componentsynchronization operations according to an embodiment of the presentinvention. Operation 1200 begins with the head end 102, the downstreamtransmitter hub 108, and the plurality of CMs 110 establishing frequencylock with the common frequency reference (step 1202). Next, operationincludes transmitting a plurality of time stamps from the head end 102to the plurality of CMs 110 via the packet data network, the downstreamtransmitter hub, and the cable modem network plant 210 (1204). Operationcontinues with each of the plurality of CMs performing smoothingoperations on the plurality of time stamps to establish phase lock withthe head end (1206). Finally, operation concludes with performingranging operations by the downstream transmitter hub 108 and theplurality of CMs 110 to establish phase lock among the plurality of CMs110 (step 1208). The ranging operations may include adjusting countersof at least some of the plurality of CMs 110.

With the operations 1200 of FIG. 12, the time stamps are distributed toall CMs 110 over the standard packet data network 202, e.g., a GigabitEthernet network. In one example, the Gigabit Ethernet network has 1 msof jitter. When averaging 10⁴ time stamps at each CM 110, each CM 110can reduce the jitter by sqrt(10⁴)=100 times, assuming independentjitter samples. This reduces the jitter to 1 ms/100=10 μs. After the CMs110 have completed this smoothing, the CMs 110 load their time counterswith the smoothed time stamp value and begin counting from there usingthe distributed frequency reference. After this has been done by all ofthe CMs 110, the following situation is in place: (1) all CMs 110 havetime counters that are off by 10 μs (1 sigma) from each other, and areincrementing their counters in lock step, using the distributed clock,so that their time offsets from each other remain constant. Thus, thesmoothing operations cause the plurality of cable modems to have counterphase offsets from one another.

The ranging operations substantially remove the phase offsets among theplurality of CMs 110 and the system is able to operate normally. Thecounters of some or all of he CMs 100 may be adjusted to remove theranging offsets. In any event, the CMTS will compensate for the offsetsin the ranging process. We have now succeeded in synchronizing all theCMs 110 using “in-band” time stamps sent over a jittery network,supplemented by a distributed frequency reference. The key idea is thatthe frequency reference enables all CMs 110 to advance their timingphase in lock step. The only challenge is their initial offsets, whichcan be solved by in-band time stamp distribution over the jittery packetdata network 202, coupled with a reasonable amount of smoothing (100× inthis example) by an amount sufficient to enable ranging to occur.

FIG. 13 is a flow chart illustrating distributed CMTS componentsynchronization operations according to another embodiment of thepresent invention. Operation 1300 begins with the head end 102, thedownstream transmitter hub 108, and the plurality of CMs 110establishing frequency lock with the common frequency reference (step1302). Next, operation includes the head end 102, the downstreamtransmitter hub 108, and the plurality of CMs 110 waiting for markersequences within the frequency reference (step 1304) and theperiodically extracting received marker sequences from the frequencyreference (1306). Operation continues with the head end, the downstreamtransmitter hub, and the plurality of cable modems setting respectivecounters based upon contents of the marker sequence (1306). Finally,operation concludes with performing ranging operations by the downstreamtransmitter hub 108 and the plurality of CMs 110 to establish phase lockamong the plurality of CMs 110 (step 1308). The ranging operations mayinclude adjusting counters of at least some of the plurality of CMs 110.

With the method 1300 of FIG. 13, a marker sequence occasionally augmentsthe distributed frequency reference. In one particular operation, a10.24 MHz frequency reference is distributed from a server to multipleclients over fiber or twisted-pair cables. The clock may be a bandlimited square wave, for example. Regularly, e.g., once per second orsimilar, the frequency reference is modulated with a short known markersequence, e.g., a 13-bit Barker sequence, or other preamble-typesequence (length less than or equal to 64 is adequate). The clients(head end 102, downstream transmitter hub 108, and/or CMs 110) are eachequipped with PLLs that track the frequency reference. These PLLs aredesigned with a narrow loop bandwidth and are therefore not adverselyaffected by the marker sequence. The PLLs may be designed specificallyto ignore the marker sequence, for example, by gating their errorsignals to zero during the time when the marker is expected to bepresent in the frequency reference. Because of this, the PLLs continueto track the frequency reference as before, with little perturbationfrom the marker.

The clients are also equipped with marker detection circuitry thatdetects the presence and time of arrival of the marker sequence. Thetime of arrival information is used to set and maintain the client'stime stamp counter. In this manner, all clients are given not only thefrequency reference, but a time reference as well. However, the markeronly comes periodically, e.g., every second. Thus, the counters can onlybe set to proper positions every one second or so, and the clientscannot determine what second it is, or any higher time unit such asseconds, minutes, hours, days, years, etc. Thus, higher time units aresent to the client over a standard network, such as the GigabitEthernet. The jitter in the Gigabit Ethernet network is no longer afactor, as long as it is much less than one second, since the finetiming information is provided by the marker, not the Gigabit Ethernetnetwork. We now have a situation where all clients are synchronized bymarker pulses. Even after these operations, varying cable lengths of thecable modem network plant 210 between the downstream transmitter hub 108to each CM 100 causes offsets in the CMs' 110 counter times(approximately 1.5 ns per ft). The ranging operations, e.g., DOCSISranging operations compensate for these offsets.

As an enhancement to the operations 1300 of FIG. 13, in addition to themarker sequence, the frequency reference may also include time stampinformation, including seconds, minutes, hours, days, years, etc. Whilethese operations may cause more interruption of the PLLs of the CMs 110to receive this time stamp information, such operations reduce theamount of information that must be sent over the packet data network 202for time synchronization.

As one of average skill in the art will appreciate, the term“substantially” or “approximately,” as may be used herein, provides anindustry-accepted tolerance to its corresponding term. Such anindustry-accepted tolerance ranges from less than one percent to twentypercent and corresponds to, but is not limited to, component values,integrated circuit process variations, temperature variations, rise andfall times, and/or thermal noise. As one of average skill in the artwill further appreciate, the terms “communicatively coupled” or“operably coupled”, as may be used herein, includes direct coupling andindirect coupling via another component, element, circuit, or modulewhere, for indirect coupling, the intervening component, element,circuit, or module does not modify the information of a signal but mayadjust its current level, voltage level, and/or power level. As one ofaverage skill in the art will also appreciate, inferred coupling (i.e.,where one element is coupled to another element by inference) includesdirect and indirect coupling between two elements in the same manner as“operably coupled.” As one of average skill in the art will furtherappreciate, the term “compares favorably,” as may be used herein,indicates that a comparison between two or more elements, items,signals, etc., provides a desired relationship. For example, when thedesired relationship is that signal 1 has a greater magnitude thansignal 2, a favorable comparison may be achieved when the magnitude ofsignal 1 is greater than that of signal 2 or when the magnitude ofsignal 2 is less than that of signal 1.

The invention disclosed herein is susceptible to various modificationsand alternative forms. Specific embodiments therefore have been shown byway of example in the drawings and detailed description. It should beunderstood, however, that the drawings and description thereto are notintended to limit the invention to the particular form disclosed, but onthe contrary, the invention is to cover all modifications, equivalents,and alternatives falling within the spirit and scope of the presentinvention as defined by the claims.

1. A method for synchronizing operation of distributed cable modemtermination system components, the method comprising: operating a headend, a downstream transmitter hub, and an upstream receiver hub, eachbeing components within the distributed cable modem termination systemcomponents, and also operating a plurality of cable modems forestablishing frequency lock with a common frequency reference;transmitting a plurality of time stamps from the head end to theplurality of cable modems via a packet data network, the downstreamtransmitter hub, and cable modem network plant; operating each of theplurality of cable modems for performing smoothing operations on theplurality of time stamps to smooth jitter incurred within the pluralityof time stamps during transmission via the packet data network and toestablish phase lock with the head end; and cooperatively performingranging operations by the downstream transmitter hub, the upstreamreceiver hub, and the plurality of cable modems to establish phase lockamong the plurality of cable modems and to synchronize clocksrespectively included within each of the downstream transmitter hub theupstream receiver hub, wherein ranging operations as performed by theplurality of cable modems involving processing first ranging burstsreceived from the downstream transmitter hub and ranging operations asperformed by the upstream receiver hub involving processing secondranging bursts received from the plurality of cable modems.
 2. Themethod of claim 1, wherein the ranging operations include adjustingcounters of at least some of the plurality of cable modems.
 3. Themethod of claim 1, wherein: the smoothing operations cause the pluralityof cable modems to have phase offsets from one another; and the rangingoperations substantially remove the phase offsets among the plurality ofcable modems.
 4. The method of claim 1, wherein the common frequencyreference is a 10.24 MHz clock.
 5. The method of claim 4, whereinestablishing frequency lock with the common frequency referencecomprises locking Phase Locked Loops of the plurality of cable modems tothe common frequency reference.
 6. A distributed cable modem terminationsystem, comprising: a head end; a downstream transmitter hubcommunicatively coupled to the head end by a packet data network; and anupstream receiver hub communicatively coupled to the head end by thepacket data network; and wherein: a plurality of cable modems beingcommunicatively coupled to the downstream transmitter hub and theupstream receiver hub by cable modem network plant; the head end, thedownstream transmitter hub, the upstream receiver hub, each beingcomponents within the distributed cable modem termination systemcomponents, and the plurality of cable modems operative to establishfrequency lock with a common frequency reference; the head endtransmitting a plurality of time stamps to the plurality of cable modemsvia a packet data network, the downstream transmitter hub, and cablemodem network plant; each of the plurality of cable modems is operativeto performing smoothing operations on the plurality of time stamps tosmooth jitter incurred within the plurality of time stamps duringtransmission via the packet data network and to establish phase lockwith the head end; the plurality of cable modems, the upstream receiverhub, and the downstream transmitter hub are cooperatively operative toperform ranging operations to establish phase lock among the pluralityof cable modems and to synchronize clocks respectively included withineach of the downstream transmitter hub the upstream receiver hub; andranging operations as performed by the plurality of cable modems involvethe plurality of cable modems processing first ranging bursts receivedfrom the downstream transmitter hub; and ranging operations as performedby the upstream receiver hub involving processing second ranging burstsreceived from the plurality of cable modems.
 7. The distributed cablemodem termination system of claim 6, wherein the ranging operationsinclude adjusting counters of at least some of the plurality of cablemodems.
 8. The distributed cable modem termination system of claim 6,wherein: the smoothing operations cause the plurality of cable modems tohave phase offsets from one another; and the ranging operationssubstantially remove the phase offsets among the plurality of cablemodems.
 9. The distributed cable modem termination system of claim 6,wherein the common frequency reference is a 10.24 MHz clock .
 10. Thedistributed cable modem termination system of claim 6, wherein eachcable modem includes a Phase Locked Loop (PLL), and the cable modems areoperative to frequency lock their PLLs with the common frequencyreference.
 11. A method for synchronizing operation of distributed cablemodem termination system components, the method comprising: operating ahead end, a downstream transmitter hub, and an upstream receiver hub,each being components within the distributed cable modem terminationsystem components, and also operating a plurality of cable modems forestablishing frequency lock with a common frequency reference; operatingthe head end, the downstream transmitter hub, the upstream receiver hub,and the plurality of cable modems for periodically extracting markersequences from the common frequency reference; upon receipt of a markersequence, operating the head end, the downstream transmitter hub, andthe plurality of cable modems for setting respective counters based uponcontents of the marker sequence; and cooperatively performing rangingoperations by the downstream transmitter hub and the plurality of cablemodems to establish phase lock among the plurality of cable modems andto synchronize clocks respectively included within each of thedownstream transmitter hub the upstream receiver hub, wherein rangingoperations as performed by the plurality of cable modems involvingprocessing first ranging bursts received from the downstream transmitterhub and ranging operations as performed by the upstream receiver hubinvolving processing second ranging bursts received from the pluralityof cable modems.
 12. The method of claim 11, wherein the rangingoperations include adjusting counters of at least some of the pluralityof cable modems.
 13. The method of claim 11, wherein the marker sequencecomprises a Barker sequence.
 14. The method of claim 11, wherein thecommon frequency reference is a 10.24 MHz clock.
 15. The method of claim11, wherein: establishing frequency lock with the common frequencyreference comprises locking Phase Locked Loops (PLLs) of the pluralityof cable modems to the common frequency reference; and the PLLs have anarrow bandwidth and do not lose frequency lock with the commonfrequency reference upon receipt of the marker sequence.
 16. The methodof claim 11: wherein establishing frequency lock with the commonfrequency reference comprises locking Phase Locked Loops (PLLs) of theplurality of cable modems to the common frequency reference; and furthercomprising gating the PLLs during receipt of the marker sequences.
 17. Adistributed cable modem termination system comprising: a head end; adownstream transmitter hub communicatively coupled to the head end by apacket data network; and an upstream receiver hub communicativelycoupled to the head end by the packet data network; and wherein: aplurality of cable modems being communicatively coupled to thedownstream transmitter hub and the upstream receiver hub by cable modemnetwork plant; the head end, the downstream transmitter hub, theupstream receiver hub, each being components within the distributedcable modem termination system, and the plurality of cable modemsoperative to establish frequency lock with a common frequency reference;the head end, the downstream transmitter hub, the upstream receiver hub,and the plurality of cable modems operative to periodically extractmarker sequences from the common frequency reference and to setrespective counters based upon contents of the marker sequence; theplurality of cable modems, the upstream receiver hub, and the downstreamtransmitter hub are cooperatively operative to perform rangingoperations to establish phase lock among the plurality of cable modemsand to synchronize clocks respectively included within each of thedownstream transmitter hub the upstream receiver hub; and rangingoperations as performed by the plurality of cable modems involve theplurality of cable modems processing first ranging bursts received fromthe downstream transmitter hub; and ranging operations as performed bythe upstream receiver hub involving processing second ranging burstsreceived from the plurality of cable modems.
 18. The distributed cablemodem termination system of claim 17, wherein the ranging operationsinclude adjusting counters of at least some of the plurality of cablemodems.
 19. The distributed cable modem termination system of claim 17,wherein the marker sequence comprises a Barker sequence.
 20. Thedistributed cable modem termination system of claim 17, wherein thecommon frequency reference is a 10.24 MHz clock.
 21. The distributedcable modem termination system of claim 17, wherein: establishingfrequency lock with the common frequency reference comprises lockingPhase Locked Loops (PLLs) of the plurality of cable modems to the commonfrequency reference; and the PLLs have a narrow bandwidth and do notlose frequency lock with the common frequency reference upon receipt ofthe marker sequence.
 22. The distributed cable modem termination systemof claim 17: wherein establishing frequency lock with the commonfrequency reference comprises locking Phase Locked Loops (PLLs) of theplurality of cable modems to the common frequency reference; and furthercomprising gating the PLLs during receipt of the marker sequences.